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CONSISTENT USER INTERFACE FRONT END FOR REMOTE USER INTERFACES 



Hie present invention relates to a system and method for a consistrait front end for 

S remote and local user interfaces. 

Throughout the foHowing text the Glossary and Abbreviations at the end of the 
description are employed, usually without reference to same. It is to be understood that these 
terms are used in the sense defined herein if there is a conflict between the ordinary meaning 
of diese terms and the terms defined herein. 
10 User interfeces are playmg an increasingly important role in home networking. In the 

area of universal plug-and-play (UPnP). however, ahnost no standards have been ather 
developed or proposed concerning user interfaces. Developers of user interfece (UI) devices, 
consequently have complete freedom to create appHcations and user interfeces for controlling 
services in a home network. Controlled devices, such as an audio device or a DVD+RW 
15 device, currentiy can optionally provide an HTML-page as a user interfece. Except using 
optional HTML-pages they have, in fact, very limited control over the appearance of a user 
interfece that is used on a UI device and can never be sure that their distinguishing features are 
accessible from these UI devices. Furthennore, some home networking scenarios, such as 
session migration and the use of multiple output devices simultaneously, are difficult to reaUze 
20 if the ownership of appUcations and user interfeces belong solely to UI devices. 

Remote UI technology is becoming an important technology in this area (next to local 
UIs) m order to deal with these limitations. PC manufecturers use it to allow appUcations 
running on a PC to be used on thin^jUent devices. Remote UI technology also enables a 
controUed device, such as a DVD+RW, to offer applications and UIs together with its services 
25 to other devices in a home network. TTus provides a way for developers to maintain 
ownership of user interfaces and appUcations since, it is important for participants to be able to 
distinguish themselves from others. Remote UI technology may also reduce the costs for 
upgrading UI devices. Therefore, it is Ukely that the various developers wiU want to have 
control over both appUcations and their user interfeces. and that multiple scenarios of remote 
30 UI ownership wiU co-exist, drivenby the marketplace for this technology. 
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Dealing wilh user interfeces in a home network is a very complex issue. Because of 
the large number of developers participating in this technology, very flerible solutions and 
more open devices ar« needed in order to meet all their requirements, such as providing a good 
out-of-the-box experience, being able to handle all kinds of (fixture) devices, services, 
applications, different configurations, and multi-user scenarios. 

The pair of usage scenarios contained in Appendix A has been promulgated as a basis 
for developing UPnP remote I/O standards and is typical of home network usage. Considering 
these usage scenarios as a starting point, the following requirements are important fat user 
into&ces in a home network. 

. It must be possible to make use of the fiinctionaUty that is offered by the services, 
devices, and appUcations withm a home network via UI devices, preferably using a 
UI with a developer-spedfic look & fijel. Generic UI devices must provide support 
far the majority of services/devices/appUcations that are available in a home 
network. 

. UI devices can have varying ciqsabilities, such as screen size, I/O modaUties, etc. 
User interfiwes must be tailored as much as possible to the specifics of the UI 
device, instead of a fi«e-floating cursor (Uke in mouse-based systems) on a TV 
using a standard remote control, and not requiring extensive use of scrolling on a 
small-screen display. Furthermore, devices can be very resource constrained. 

• In order to be successfiil, home network products must not be more difiBcult to use 
than non-networiced products. UIs must automatically deal with network 
configuration changes. In addition, it is important that UIs be consistent (for 
similar controUed devices as weU as among UI devices from different vendors). 
This means, for example, that: 

a) preferably, aU user mterfiices on a generic UI device have the same look & feel, 
for example, changing volume is done with the same UI for every device and 
not by having a con«)letely different UI for every developer (e.g. a sUder with 
label **Volume" in one case and a rotating knob with label "VoL Level" in 
anotho: case), and 
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b) the interaction with a particular device/service/appUcation must be similar 
among all the UI devices (e.g. not show "Play tcom DVR" on one UI device. 
•Thiy fiom HD" on a another device and 'TPlay MPEG-2 files" on a PC. 
because this might create confusion and annoyance among users. 
5 . The cost for adding networking capabiUties to devices must be kept at a minimum. 

In addition to the scenarios of Appendix A. the following scenarios are important scenarios 
that networked user interfece technology must be able to support: 

a) ^^n ^ tn rnntent r.^fml nf devices: It must be possible for the user to 
have access to the content that is available in the network as well as the abiUty 
,0 to control devices and services, fiom any compatible device anywhere inside 

tiie home, and in a coherent way. It must be possible, for example, to 
browse/search throu^ all the content as if it were a single database (at least per 
category, such as through all MP3s), and rendo: the desired content on a user- 
specified compatible rendering target Besides content-browsing and device 
,5 control appUcations, networked user mlerfaces must also be able to support 

communication-based appUcations, commerce, information services, games, 
productivity applications, and so on. 
b) n»t-nf-the-hox experience : It must be easy for the user to add new devices to 
the home network. Example: User X has just purchased a generic UI device to 
20 control his home network. His home network consists of several UPnP devices. 

After switching on the control device it shows him that it has successfiiUy 
detected several CE devices and that it has automatically downloaded the 
newest drivers for most of them. Unfortunately, one of them is not supported. 
The control device support center has been contacted about the problem and the 
25 user wiU be notified automatically when an update is available. In the 

meantime, flie user is advised to download and install some bridging software 
fiom the control device support center to the user's PC. with which the user can 
create simple HTML pages from service descriptions for use in the mean-time. 
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c) Multifile users ; Typically, a home network is used by more flian one person. 
Dififei«nt femily members may want to use the same 
devices/services/appUcations of an in-home network at the same time. 
Therefore, it must be easy to switch to an alternative service or service instance, 
or to another UI device. It is desirable to share the same resource (e.g. an 
application) with other people. For example, it must be possible to play a game 
against each other, where each player has thdr own UI device and their own 
view of the same application. Synchronization is very important in such 
applications in a home network. 

d) P^^o pali^ation : The settings of one user are probably not appUcable to other 
users (e.g. musical preferences in an mp3-jukebox appUcation). Furthermore, 
there is a need for communication among users. For example, send a "dinner is 
ready" message to everybody using the home network. 

e) Mnirt ple T/Q devices; It must be possible to use multiple input and multiple 
output (i.e. multiple UI) devices for ttie same appUcation at the same time, to 
enable UIs where the graphical part of the UI is shown on the TV and the 
audible part of the UI is redirected to the speakers that are closest to the user. 
Or, to enable appUcations such as an inteUigent photo browser with which the 
user can redirect pictures fiom a mobile device to a bigger screen. 

Thus, there is a need for consistency of user interfeces across consumer electronic (CE) 
devices tiiat are resident in a home network. 

The present invention provides a system and method that uses an extensile database 
of "synonyms" containing constructs of a UI that can be used to replace correspondingly 
similar UI constructs (also caUed consents) of a CE device, such as labels of UI elements 
25 and common user interface patterns, wherein a CE device can make incoming remote UIs 
more consistent with one another. In a preferred embodiment of a home network, one CE 
device is typicaUy a UI control device for aU the CE devices in the network. On power-up. 
each of the CE devices sends its UI to the control CE device and it is replaced with a 
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consistent UI, according to a prefened embodiment, whenever the control device presents that 
Ultoauser. 

The database of "synonyms" is used by the control CE device by examining each 
incommg remote UI for constructs and matching these found UI constructs with known similar 
5 "synonyms" in the database. OptionaUy, those found UI constructs that are not already in the 
database can be added to the database during this process (by the CE device locally and by 
downloading the found constructs to a CE device service center). A basic set of these 
"synonyms" is provided to initiate this system and method. 

Note: the above described usage of a database of "synonyms" is not to be confused 
10 with using synonyms in speech where the intent is to ext«id the expressiveness of language by 
introducing equivalent terms. Here, the intent is to limit the variety of expression to a single 
consistent set of constracts based on equivalency of a variety of temis. 

The foregoing dnd other features and advantages of the invention wiU be parent fixmi 
the following, more detailed descriptibn of preferred embodiments as iUustrated in the 
15 accompanying drawings in which reference characters refer to the same parts throughout the 
various views. 

FIG. I iUustrates the process of finding and substituting synonyms for remote UI 
components according to the present invention. 

FIG. 2 illustrates an example of hardware components of a control device within a 
home network that can be used to perform the present invention. 

HG. 3 iUustrates a typical home network with a control CE device and slave CE 
devices whereto embodiments of the present invention can be appUed. 
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It is to be understood by persons of ordinary skiU in the art that the following 
descriptions are provided for purposes of illustration and not for limitation. An artisan 
understands that there are many variations that Ue within the spirit of the invention and the 
scope of the appended claims. Unnecessary detail of known functions and operations may be 
omitted from the current description so as not to obscure the present invention. 
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A preferred embodiment of the present invention is ilhistrated in FIG. I. in which one 
or more controlled devices can provide multiple remote user interfiice alternatives to the UI 
device 200. In order for a UI device to know which mechanisms are ofifaed by a controlled 
device (and vice versa), it must be possible to discover this information. This discovery can 
5 be accomplished, for example, by means of a Remote I/O capability description which can be 
found in a similar way as service descriptions. After this information has been retrieved, a 
protocol can be chosen that is best .suited for the atuation. 

One or more (remote) RUI definitions 101, which can be in any format, including 
RDP, X-Windows, VNC, HTTP, HAVi DDI, and UI Fragments, XForms and Java AWT. are 
10 taken (simultaneously) as input to an extraction process 102, where the relevant user interfiice 
data (i.e. assets) that makes up the user interface (such as labels, bitm^s, widgets) 120 are 
extracted from the incoming 1 10 UI definition. If the UI definition input 1 10 is defined on an 
abstract level, such as XForms, then this is accomplished in a relatively straightforward 
manner, as is known in the art For less abstract or mixed input (sudi as RDP or HTML) 
15 known techniques such as model recovery [1] are applied to extract the relevant UI data 120 
and the underlying semantic model of the XJI from the incoming concrete UI 110. 

After the extraction process 102 the user interfece data is analyzed and transformed 
(syntactically or semantically) 103 using a database of "synonyms" 104: 
ORIGINAL gyWONYM 
20 a, a', a", a'" a 

b,b',b" b' 

X X 

y,y* y' 

25 If a match is found, and analysis of the user interface, with user preferences taken into 
consideration 107. shows that this replacement will indeed improve the user interface (e.g. by 
calculating a measure of confidence, and, optionaUy. through learning from previous analyses 
and transformations), then the preferred "synonym" replaces the original input 120 and the 
modified UI information 121 is output, which can be used to render the UI 106: 

30 
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UI MODIFIED UI 
label a' • 

label b" label b' 

label c Jabelc 
5 widget X widget X 

widget y widget y' 

• • • 

• • • 

If no match is found, the system connects to the Internet 105 to check on-Une thesauri 109 or 
other relevant databases. If still no match is found, the original value remams in the UL 
10 The preferred "synonym" which is used for transformation can dther be a defiiult value 

(e.g. the first in the list) or one which is selected by a user 108. In a preferred embodiment, 
the user is allowed to add "synonyms", for example, to give names to devices and processes in 
a home network by inputting preferences 107. After this transformation step 103, the 
modified Ul 106 is either shown to the user 108 or transferred to another device. In the first 
15 case, the synonym transformation process is done on a UI device. In the latter case, the 
synonym transformation process is offered as a service in the network. 

Refrarihg to FIG. 2. the control CE device within the home network of HG. 3 may 
inchide a system with an architecture that is illustrated in the block diagram of FIG. 2. On 
power up. each slave CE device 101 transmits its RUI 110 to the control CE device 200 for a 
20 consistent presentation of its UI to a user by the control CE device 200. The control CE 
device 200 may include a transceiver 203, an extraction logic module 102, an analysis and 
transformation module 103. a synonym database 104. a memory 205 and a timer 204. The 
exemplary system 200 of FIG. 2 is for descriptive purposes only. Although the description 
may refer to terms commonly used in describing particular control CE devices, the description 
25 and concepts equaUy apply to other processing systems, including systems having 
architectures dissimilar to that shown in FIG. 2. 

In operation, the transceiver 203 is coupled to an antenna (not shown) and to a wireUne 
to transmit and receive data to and fi»m wireless and wired slave CE devices 101.2 and 101.1, 
respectively. The extraction logic module 102 extracts components of a received slave device 
30 UI definition. After the extraction process 102 the extracted UI data 120 is analyzed and 
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transfonned by an analysis and transfonnation module 103 to replace components of the 
received slave device UI 120 with matching synonyms from the synonym database 104 to 
produce a transfonned XJI definition 121, and is displayed on the user mterfiice module 207 as 
the interfece to the remote (slave) device or is transmitted by transcdver 203 to a second (or 
third, etc.) control device (not shown). The memory 205 is used by both the extraction logic 
module and the analysis and transformation module 103 to store learning derived from 
previous processing of slave CE device UIs and optionally update the synonym database 104 
therewith. The timer 204 is used by the logic modules for time dependent processes, such as 
outdated transformed UIs for slave CE devices that have powered down. 

FIG. 3 iUustrates a representative home network whereto embodunents of the present 
invention are to be appUed. As shown in FIG. 3, a contix)l CE device 200 is coupled to a 
plurality of slave wired and wireless CE devices 101, which, tiiiough tiie wireline and wireless 
links of a home network 206 communicate their UIs to tiie control CE device 200. A key 
principle of tiie present mvention is to provide a mechanism to substitiite components of a 
consistent user interface for similar components in each slave CE device UI 101. The contirql 
CE device 200 performs an exti^tion of user interface components and tiien analyzes and 
transforms 103 tiiem to finally yield a transformed UI 106 for each slave CE device 101. 
Alttiou^ a limited number of CE devices is shown in FIG. 3 for iUustrative purposes, it is to 
be undwstood fluit the home network can support communications between a much larger 
number of CE devices. Thus, tiie number of CE devices in FIG. 3 should not impose 
limitations on tiie scope of tiie invention. Further, tiie home network can be based on any of 
internet protocol (IP) (RFC 791). NETBEUI, Bluetootii, Zigbee. SCP, IEC61883, DVB and 
ATSCDTV. 

In replacing components of tiie remote user interfece, one or more contiwl devices can 
be employ different replacement paradigms comprising predetermined similarity measures 
between conqionents of ttie GUI and tiie RUI. predetermined improvement measures in tiie 
consistency of flie RUI after replacement of an RUI component by a CUI component (tiiis can 
be a cumulative measure depending on flie number of such replacements or mappings), 
predetermined difference measures between conq»onents of tiie CUI and tiie RUL 
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In an alternative embodiment, the control device also has one or more local UIs 
(LUIs), i.e. a user interfece for a local appUcation (see Glossary term "AppUcation"). that it 
can display, in addition to displaying RUIs. Since these LUIs may contain some user interfece 
constracts (components), which are similar to constructs (components) used in an incoming 
RUI. it is useful to make the incoming RUIs not only consistent with each other but also with 
LUIs (so if the label "Play from DVR" is flie preferred synonym, then this label has to be used 
in the RUIs as weU as the LUIs). In a preferred embodiment, the synonym that is chosen in 
the transformation process 103 is the synonym tiiat is used in an LUI, in order to make the 
incoming RUIs more consistent with LUIs of a control device. 

WhUe the preferred embodiments of the present invention have been illustrated and 
described, it will be understood by those skiUed in the art that various changes and 
modifications may be made, and equivalents may be substituted for elements thereof without 
departing from the true scope of the present invention. In addition, many modifications may 
be made to adapt to a particular situation and the teaching of tiie present invention can be 
adapted in ways that are equivalent wittiout departing from its central scope. Therefore it is 
intended that the present invention not be limited to the particular embodiment disclosed as the 
best mode contemplated for carrying out the present invention, but that the present invention 
include all embodiments felling within tiie scope of the appended claims. 
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Claims: 

1. A method for a consistent user interface (GUI) on a control device providing 
access to at least one network device having a remote user interfece (RUI), comprising the 

steps of: 

providing the GUI; 

mapping tiie RUI to the CUl; and 

displaying hy the control device at least a part of the GUI instead of the RUI as a user 
interface to the network device. 

2. The method of claim 1, wherein: 

the providing step furthra: comprises the steps of: 

receiving by the control device an RUI definition comprising at least one RUI 

component; 

providmg the GUI comprising at least one GUI component optionally pre-set as 
a synonym for said at least one RUI component; and 

the mapping step further comprises the step of replacing the received at least one RUI 
component with said pre-set synonym GUI component by the control device whenever the 
control device displays a user interfiace to the networic device. 

3. The method of claim 2, further comprising the step of transmitting the RUI 
definition by the at least one network device. 

4. The method of claim 3, wherein said transmitting step further comprises the 
step of on power-up. transmitting the RUI definition by the at least one network device, 

5. The method of claim 3, wherein said transmitting step further comprises the 
step of transmitting the RUI definition using a network based on at least one of IP (RFG 791). 
NEIBEUI, Bluetooth. Zigbee, SGP, IEG61883, DVB and ATSG DTV. 
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6. The method of claim 5, wherein said transmitting step further comprises the 
step of transmitting the RUI definition using a protocol based on at least one of RDP. X- 
Windows, VNC, HTTP, HAVi DDI. and UI Fragments. 

7. The method of claim 5, wherein said receiving step further comprises the step 
of using by the at least one network device for the RUI definition a Ul description format 
based on at least one of HTML, XML. Macromedia. Flash and Java. 

8. The method of claim 3, further comprising the step of sending the provided 
cm to at least one of the at least one network device and a second control device. 

9. The method of claim 7, further comprising the step of transmitting aRUI 
definition by at least a second networic device to the control device using at least one of a 
different protocol selected from the set consisting of RDP, X-Windows, VNC, HTTP, HAVi 
DDI. and UI Fragments and a different UI description format selected from the group 
consistmg of HTML. XML, Macromedia. Flash and Java than the corresponding protocol and 
description format used by the at least one network device. 
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10. The method of claim 1, whaein: 

the RXn and CUI comprise at least one RUI component and at least one GUI 
component, respectively, and 

further comprising the steps of: 

extracting said at least one RUI component; 

determining if said at least one CUI component is a synonym for the extracted at least 
one RUI component; and 

wherein, said mapping step further comprises the step of 

if said at least one CUI component is detemiined to be a synonym for the extracted at 
least one RUI component, mapping the at least one extracted RUI component to &e 
detemiined said at least one CUI componait. 

11. The method of claim 10. further comprising the step of finding at least one CUI 
component that satisfies a predetermined similarity measure to said at least one RUI 
component for a pluraUty of network devices. 

12. The method of claim 10. further comprising the step of finding at least one CUI 
component that satisfies a predetemiined consistency measure of the mapped at least one RUI 
componCTt 

13. The method of claim 10, wherein said determining step further comprises the 
step of searching at least one of a thesaurus and a synonym database for a synonym of the 
extracted at least one RUI component that matches said at least one CUI component 

14 The me&od claim 13, further comprising the step of storing said matching 
synonym detemrined from the tiiesaurus m the synonym database for the extracted at least one 
RUI component 

15. The method of claim 14 wherein said at least one thesaurus is located on a 
second network and said at least one network device is located on a first network. 
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16. The method of claim 15, wherein said second network is the Internet 

17. The method of claim 15, wherein said first network is a home network and said 
network device is a consumer electronic device. 

18. The method of claim 1, wherein smd at least a part of the GUI is determmed 
according to a set of user preferences. 

19. The method of claim 1 , wherein: 

said at least one network device further comprises at least one application local to ttie 
control device and said RUI further comprises at least one local user interface (LUJ) to said at 
least one local application. 

20. The method of claim 10, wherein: 

said at least one network device further comprises at least one application local to the 
control device and said RUI further comprises at least one local user interface (LUI) to said at 
least one local application. 

21 . A method for a slave network device to replace a remote user interface (RUI) 
with a consistent user interface (CUT), comprising tiie steps of: 

transmitting by the slave network device the RUI to a control network device; and 
replacing by the control network device at least a part of the transmitted RIH with at 
least a part ofthe consistent GUI by the method of claim 1. 

22. A control device that provides a consistent user interface (GUI) in a network of 
at least one slave device having a remote user interface (RUI), comprising: 

a transceiver for receiving the RUI; 

an extraction logic module configured to extract at least one component ofthe RUI; 
a database fliat is configured to store synonyms of components of an RUI; and 
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an aiialysis and traiisfearmation module configured to 

i. map the extracted at least one component of the RUI to a component of the CUI 
according at least one of the synonyms stored in said database and a thesauri. 

ii. store the mapping in a memory, 

iii. optionally update the synonym database with the moping, and 

iv. provide a user interfece to the at least one slave device according to at least a 
part of the mapping of the extracted at least one component of the RUL 

23. The control device of claim 22, further comprising: 
at least one local user interface (LUI) to at least one local ^Ucation; 
wherein, 

said extraction logic module is further configured to extract at least one component of 
the LUI; 

said database is fiirther configured to store synonyms of components of an LUI; 

said analysis and transformation module is further configured to 

V. map the extracted at least one component of the LUI to a component of the CUI 

according to at least one of the synonyms of components of an LUI stored in said database and 

adiesauri, 

vi. provide a user interface to the at least one local appUcation according to at least 
a part of the mapping of the extracted at least one component of the LUL 

24. The control device of claim 22, wherein said analysis and transformation 
module is further configured to: 

vii. accept a set of user preferences; and 

viii. provide said at least a part of the mapping in accordance with the set of user 



25. The control device of claim 22, wherein the thesauri is accessed via another 
netwoik. 
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26. The control device of claim 25. wherein the another network is the Intern 

27. The control device of claim 22, wherein the slave device is a consumer 
electromc device. 

28. The control device claim 27, wherein the network is a home network. 

29. The control device of claim 22, wherdn the network is a home netwoik. 
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ABSTRACT 



A method and device for providing a single consistent user interface based on 
replacing disparate user interface constmcts (components) with equivalent constructs 
(components) fimn a "synonym" database of user interfece constructs (components). The 
database is initiated with a set of foundational "synonym" constructs (components) and 
extended with user interfece constructs (components) as they are discovered during reduction 
of an inconsistent user interfece to a consistent state. 
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